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I .   INTRODUCTION 

A.   OVERVIEW 

Organizations  of  all  types  are  seeking  to  improve  their 
effectiveness  and  efficiency  by  using  information  systems. 
Considerable  amounts  of  time  and  money  are  spent  to  develop 
and  implement  information  systems  within  an  organization. 
Once  inplemented,  an  information  system  may  or  may  not  be 
effective  in  accomplishing  its  objective.  Thus,  an  important 
and  necessary  stage  in  justifying  any  particular  information 
system  is  to  determine  its  value  of  effectiveness  to  the 
organization . 

The  Department  of  Defense  (DoD)  has  undertaken  the 
development  and  implementation  of  a  large  information  system 
for  use  at  Medical  Treatment  Facilities.  The  system  is  called 
the  Composite  Health  Care  System  (CHCS)  .  Due  to  the  costs 
associated  with  this  system,  a  simple  and  efficient  measure  is 
needed  to  analyze  the  effectiveness  of  this  information 
system. 

Selecting  the  appropriate  measure  of  system  effectiveness 
is  difficult.  Much  emphasis  is  placed  in  many  organizations 
on  economic  analysis  of  information  system.  This  type  of 
analysis  provides  only  an  economic  measure  of  effectiveness 
for  a  system:  will  the  benefit  outweigh  the  cost?   If  the 


objective  of  the  organization  is  not  purely  economic,  the 
measure  is  not  complete. 

Many  researchers  have  proposed  measuring  the  level  of 
satisfaction  perceived  by  the  users  of  the  system  as  a 
surrogate  measure  of  effectiveness.  No  matter  how  technically 
correct  an  information  system,  if  the  users  are  not  satisfied 
with  the  system,  it  is  not  an  effective  system.  Conversely, 
if  the  users  are  satisfied,  the  system  is  considered  effective 
in  meeting  their  needs. 

B.   RESEARCH  OBJECTIVES 

This  study  will  investigate  through  empirical  research 
whether  the  Composite  Health  Care  System  is  effective  through 
the  use  of  the  user  information  satisfaction  surrogate 
measure.  Using  a  previously  developed,  documented,  and 
validated  survey  instrument,  this  study  will  describe  the  user 
information  satisfaction  characteristics  at  one  implementation 
site  of  the  Composite  Health  Care  System:  Naval  Hospital, 
Charleston.  The  results  of  this  survey  will  help  pinpoint 
potential  problem  areas  with  the  system  implementation  as  well 
as  document  areas  without  problems.  Once  completed,  the 
results  will  serve  as  a  baseline  gauge  to  compare  satisfaction 
with  CHCS  at  other  facilities  and  to  measure  changes  in 
satisfaction  at  Charleston  at  a  later  time. 


II.   DESCRIPTION  OF  THE  COMPOSITE  HEALTH  CARE  SYSTEM 

A.  BACKGROUND 

The  Department  of  Defense  (DoD)  has  pursued  the  goal  of 
providing  automated  computer  support  to  its  hospitals  and 
clinics  since  1968.  In  February  1979,  the  Mission  Elements 
Need  Statement  (MENS) ,  establishing  the  need  for  an  automated 
computer  system  was  approved.  Under  the  direction  of  the  Tri- 
Service  Medical  Information  System  (TRIMIS)  program  office, 
stand-alone  and  integrated  health  care  computer  systems  were 
acquired,  implemented  and  operated  to  support  Pharmacy 
(TRIPHARM) ,  Laboratory  (TRILAB) ,  Radiology  (TRIRAD) , 
Appointment  and  Scheduling  (TRIPASS) ,  Quality  Assurance 
(AQCESS) ,  and  Hospital  Information  System  (HIS) .  The 
knowledge  and  experience  gained  through  the  operation  of  these 
systems  since  197  9  was  used  to  refine  and  validate  the  system 
requirements  for  a  completely  new  and  fully  integrated 
information  system:  The  Composite  Health  Care  System  (CHCS) . 

B.  CHCS  PURPOSE 

The  primary  purpose  of  CHCS  is  "to  provide  health  care 
services  that  support  military  forces  in  fulfilling  their 
required  mission...."  (Draft  System  Decision  Paper,  1989). 
Improving  the  efficiency  and  effectiveness  of  military  health 
care  delivery  during  peacetime  and  during  mobilization  through 


integration  of  information  resources  is  the  goal  of  CHCS .  The 
major  focus  of  CHCS  is  the  integration  of  shared  information 
resources  to  resolve  the  deficiencies  of  communication, 
decision  support,  and  information  processing.  The  Composite 
Health  Care  System  is  designed  as  a  fully  integrated  medical 
information  system  that  provides  automated  support  of 
information  requirements  for  military  medical  treatment 
facilities.  CHCS  supports  the  administrative  functions  and 
the  delivery  of  health  care  with  information  retrieval 
services . 

C.  DEPLOYMENT  STATUS 

CHCS  is  undergoing  the  Operational  Test  and  Evaluation 
(OT&E)  phase  and  is  currently  deployed  to  Naval  Hospital 
Charleston  as  one  beta-test  site.  The  role  of  a  beta-test 
site  is  to  incrementally  replace  the  TRIMIS  systems  with 
validated  CHCS  software.  Continued  validation  of  requirements 
and  extensive  evaluation  of  the  CHCS  software  is  conducted 
prior  to  the  decision  to  deploy  CHCS  worldwide. 

D.  OPERATIONAL  FEATURES 

CHCS  supports  various  functional  areas  in  sharing 
information.  Each  functional  area  uses  this  information  for 
its  own  purpose  and  communicates  results/activities  to  other 
areas.   Functional  areas  supported  by  CHCS  include: 


1 .  Patient  Administration 

Patient  administration  does  the  registration  of  a 
patient  into  the  system.  Once  in  the  computer  system,  the 
patient's  demographic  information  is  available  for  access  by 
all  other  modules.  The  patient  administration  module  also 
performs  the  functions  to  admit  patients  to  the  hospital, 
transfer  patients  between  wards,  and  discharge  patients  from 
the  hospital.  The  creation,  update  and  closing  of  inpatient 
records  is  also  performed  by  the  patient  administration 
module . 

2 .  Patient  Appointment  and  Scheduling 

The  Patient  Appointment  and  Scheduling  module  provides 
for  a  centralized  appointment  service,  a  decentralized 
appointment  service,  or  a  combination  of  both  depending  on  the 
desire  of  the  hospital.  The  appointment  service  creates  and 
maintains  the  appointment  schedule  for  the  hospital.  A 
patient  is  given  an  appointment  date  and  time  to  see  the 
health  care  provider  creating  a  centralized  repository  of 
information  to  produce  operational  reports  of  activity. 

3 .  Nursing 

CHCS  supports  inpatient  nursing  in  a  variety  of  areas. 
In  the  area  of  patient  care,  nursing  uses  the  system  for 
entering  and  obtaining  the  status  of  all  physician  orders, 
entering  patient  assessment  data,  and  generating  patient  care 
plans.   Documentation  of  patient  progress  is  performed  in  the 


automated  nursing  notes.  Nursing  unit  management  uses  CHCS  to 
generate  shift  care  plans,  patient  management  reports,  drug 
administration  time  reports,  and  staffing  requirement  reports. 
Routine  administrative  functions  are  performed  by  CHCS  for 
nursing  including:  staff  credentialling,  documentation  of 
continuing  education  and  in-service  training,  and  staff 
scheduling  capabilities. 

4 .  Laboratory 

The  laboratory  uses  the  CHCS  system  for  processing  of 
orders  for  laboratory  tests  with  automated  reporting  of  test 
results.  Specimens  are  processed,  tracked,  and  reported 
through  the  system.  All  results  are  placed  into  the  patients 
automated  medical  record  and  are  immediately  available  for 
inquiry  by  health  care  providers.  Automated  control  of  blood 
bank  operations  including:  blood  acquisition,  inventory  and 
utilization  are  accomplished  through  CHCS.  The  laboratory  is 
supported  with  ability  to  manage  inventory  control,  register 
patients  into  the  tumor  registry,  conduct  a  drug  testing 
program,  and  generate  a  multitude  of  management  reports. 

5 .  Pharmacy 

The  pharmacy  uses  CHCS  to  process  prescriptions  for 
patients  that  are  ordered  by  health  care  providers.  The 
health  care  provider  enters  the  prescription  information  at  a 
terminal  located  in  his  office.  The  information  is 
electronically  transmitted  to  the  pharmacy.   The  pharmacy  can 


immediately  prepare  the  medication  for  dispensing  before  the 
patient  leaves  the  physician's  office.  CHCS  automatically 
performs  checks  for  patient  allergies,  drug  interactions  with 
medications  the  patient  is  currently  taking,  drug  dosage  for 
the  age  of  the  patient,  and  records  the  prescription  into  the 
patient's  medical  record.  In-patient  order  processing  is 
accomplished  in  much  the  same  fashion.  On-line  drug 
monographs  are  available  for  inquiry  by  medical  staff  at  any 
terminal.  Pharmacy  management  of  inventory  control  and 
reporting  are  also  accomplished. 

6 .  Radiology 

Orders  for  radiological  procedures  are  electronically 
transmitted  to  the  radiology  department  where  the  order  is 
processed.  Order  tracking  and  results  reporting  are  performed 
by  the  system.  Radiology  results  are  immediately  available 
for  inquiry  by  medical  staff.  Department  management  including 
inventory  control  is  also  conducted  through  CHCS. 

7 .  Clinical  Dietetics 

The  Clinical  Dietetics  service  uses  CHCS  to  obtain 
patient  data  needed  to  access  the  nutritional  needs  of  the 
patient.  This  information  is  used  to  create  patient  diet 
plans,  select  patients  of  interest,  patient  menu  selections 
and  monitor  patient  nutritional  data.  CHCS  is  also  used  to 
maintain  the  nutrition  clinic  schedule  and  compute  clinical 
dietetics  workload  data. 


E.   POTENTIAL  BENEFITS 

In  general  terms,  the  overall  benefits  of  the  CHCS  system 
include: 

1.  reliable,    timely,    easily   accessible   clinical 
information 

2 .  improved  documentation  for  medicolegal  issues 

3.  improved  communication 

4.  improved  patient  satisfaction 

5.  improved  management  efficiency 


III.   USER  INFORMATION  SATISFACTION  AND  ITS  MEASUREMENT 

A .  INTRODUCT ION 

Organizations  spend  considerable  amounts  of  time  and  money 
to  develop  and  implement  information  systems.  However,  since 
financial  resources  are  scarce  and  limited,  not  all 
application  systems  can  be  developed.  Only  those  applications 
that  can  be  justified  are  approved  and  developed.  An 
important  and  necessary  stage  in  justifying  any  particular 
information  system  is  to  determine  its  value  or  effectiveness 
to  the  organization. 

B.  METHODS  TO  ASSESS  EFFECTIVENESS 

Several  methods  have  been  suggested  in  the  empirical  and 
non-empirical  literature  for  assessing  the  effectiveness  of  an 
information  system. 

System  usage  (Swanson,  1974;  Conrath  and  Mignen,  1990) 
assumes  that  a  casual  relationship  exists  between 
effectiveness  of  an  information  system  and  the  amount  of  time 
the  system  is  used.  An  effective  information  system  is  used 
more  than  a  less  effective  system.  This  method  also  assumes 
that  the  use  of  the  system  is  optional  to  the  user.  If, 
however,  the  use  of  the  system  is  mandatory,  the  relationship 
between  use  and  effectiveness  fails. 
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Others  (eg.,  Hamilton  and  Chervany,  1981)  argue  that 
effectiveness  is  determined  by  comparing  performance  to 
ojbectives.  To  assess  the  effectiveness  of  an  information 
system,  the  task  objective  of  the  system  is  first  determined. 
Criterion  measures  are  developed  to  measure  the  extent  to 
which  the  information  system  accomplishes  the  determined 
objectives.  Problems  are  encountered  with  this  method 
because:  objectives  and  measures  are  often  not  defined 
adequately;  efficiency-oriented  and  easily  employed  measures 
are  frequently  used  over  effectiveness-oriented  and  difficult 
measures;  individual  interpretations  of  what  the  ojectives  and 
measures  are,  often  conflicting,  exist. 

Another  method  frequently  used  to  determine  the  value  of 
an  information  system  is  by  economic  (cost-benefit) 
evaluation.  The  benefits  of  a  system,  by  improved 
organizational  effectiveness,  are  weighed  against  the  cost  for 
development  and  operation  of  the  system  (Nolan,  1974) . 
Although  this  appears  objective  and  comprehensive  on  paper,  in 
practice,  cost-benefit  analysis  is  highly  subjective  and 
difficult  to  conduct  (Nolan,  1974) .  The  difficulties  in  using 
this  approach  for  research  argued  by  Ives  et  al .  (1983)  stem 
from  the  fact  that:  (1)  many  costs  and  benefits  are  intangible 
and  not  easily  recognizable  and/or  converted  into  monetary 
equivalents,  (2)  unstructured,  ad  hoc  decision  making  benefits 
are  nearly  impossible  to  objectively  assess,  (3)  even  when 
these  items  are  determined  by  an  organization,  the  data  are 
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generally  unrecorded  and  not  available  for  research  (Ives  et 
al.,  1983). 

Theoretically,  evaluation  of  the  effectiveness  of  an 
information  system  is  best  accomplished  by  decision  analysis, 
based  on  its  degree  of  use  in  decision  making  and  the 
resultant  productivity  benefits  attributable  to  its  use 
(Nolan,  1974) .  An  effective  information  system  supports  a 
user  by  retrieving  necessary  information  in  the  proper  format, 
level  of  detail,  and  at  the  proper  frequency  for  use. 
However,  the  decision  analysis  approach  suffers  a  significant 
drawback  for  use  in  scientific  research:  it  is  non- 
quantifiable  and  cannot  be  easily  replicated. 

C.   THE  USER  INFORMATION  SATISFACTION  CONSTRUCT 

A  surrogate  for  utility  in  decision  making  is  the 
satisfaction  of  users  of  the  information  system  (Neumann  and 
Segev,  1980;  Ives  et  al .  ,  1983;  Bailey  and  Peason,  1983; 
Baroudi  and  Orlikowski,  1988) . 

Webster' s  New  Collegiate  Dictionary  defines  satisfaction 
as  "the  fulfillment  of  a  need  or  want".  Organizational 
Psychologist  Bernard  Bass  (1965)  generically  defines 
satisfaction  to  mean  "...  the  extent  to  which  the  item  is 
rewarding  to  us  relative  to  how  much  better  we  might  do 
elsewhere  and  what  aspirations  we  have." 

The  concept  of  User  Information  Satisfaction  (UIS)  can  be 
traced  back  to  the  work  of  Cyert  and  March  (1963)  .   Their 
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research  argues  that  organizational  behavior  continually 
imposes  upon  the  manager  the  need  for  information.  If  a 
formal  information  system  exits,  the  success  or  failure  of 
that  information  system  to  meet  the  needs  of  the  user  either 
reinforces  or  frustrates  the  user's  sense  of  satisfaction  with 
that  system. 

Figure  3.1,  an  adaptation  of  the  Cyert  and  March  model, 
portrays  the  reinforcement /reduction  of  satisfaction  process 
in  decision  making  using  the  information  system.  When  a  user 
perceives  that  the  needed  information  should  be  readily 
available  in  the  formal  information  system,  he  will  use  the 
information  system  in  the  normal  manner  to  retrieve  the 
required  information.  If  the  information  is  readily 
available,  satisfaction  with  the  system  will  be  reinforced. 
But,  if  the  information  is  not  readily  available,  he  must 
adapt  and  conduct  a  search  beyond  the  information  system  by  ad 
hoc  methods  or  by  some  other  less  familiar  method. 

An  "expanded  search"  through  the  information  system  is  a 
cause  of  frustration  to  the  user  because  of  the  time  and 
effort  used  to  obtain  the  necessary  information. 
Additionally,  the  user  is  frustrated  because  he  perceived  the 
information  to  be  readily  available  when  it  was  not.  From 
users'  standpoint,  information  systems  may  actually  block  the 
user  from  carrying  out  the  act  of  decision  making  (Nolan, 
1965) . 
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in  the  inforamtion 
system? 


Yes 


No 
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Expand  search 
beyond  present 
information 
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Reinforcement  of 
satisfaction  with 
the  information 
system 


1 


Reduction  of 
satisfaction  with 
the  information 
system 


iz 


Make  decision 


i^z. 


Obtain  Information 


Figure  3.1 

UIS  reinforcement/reduction 

in  decision  making 


The  information  system  is  continually  being  evaluated  by 
the  user  from  repetitive  use.  If  over  a  period  of  time  of 
use,  the  user  perceives  that  the  information  system  cannot 
retrieve  the  information  without  the  frequent  use  of  expanded 
searches,  frustration  will  occur.  Consequently,  this  may  lead 
to  an  expression  of  user  dissatisfaction  with  the  information 
system.  The  project  is  a  failure  if  the  end  product  does  not 
satisfy  the  manager  whom  it  is  to  serve  (Powers  and  Dickson, 
1973) .   If,  however,  the  user  is  able  to  reliably  obtain  the 
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needed  information  with  normal  operation  of  the  information 
system,  satisfaction  with  the  system  will  be  reinforced  and 
the  information  system  is  successful. 

Of  note  are  two  empirical  studies,  (Aldag  and  Power,  1986; 
Gallupe  and  Desanctis,  1988) ,  undertaken  to  find  a 
relationship  between  satisfaction  and  actual  system 
performance  using  objective  third  party  judges.  The  results 
of  these  studies,  however,  are  conflicting  but  do  not  support 
the  assumption  that  increased  satisfaction  indicates  actual 
system  performance  increases. 

D.   THE  MEASUREMENT  OF  USER  INFORMATION  SATISFACTION 

Potentially  measurable  and  generally  acceptable,  several 
studies  have  employed  user  information  satisfaction  (UIS)  as 
a  dependent  variable  to  indicate  system  effectiveness  and 
acceptance  (Ives  and  Olson,  1979;  Igbaria  and  Nachman,  1990) . 
Neumann  and  Segev  (1980)  show  a  correlation  between  user's 
reaction  to  satisfaction  factors  and  their  perception  of  the 
organizations' s  performance .  Swanson  (1974)  empirically  found 
high  correlation  between  the  user' s  appreciation  of  the  system 
and  his/her  usage  of  the  outputs.  Powers  and  Dickson  (1973) 
concluded  that  user  satisfaction  is  the  most  critical  factor 
for  system  success.  Although  there  is  no  standard  measure  of 
satisfaction  in  these  studies,  it  is  argued  that  user 
information  satisfaction  is  an  indicator  of  system  usage  and 
success . 
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The  level  of  user  information  satisfaction  in  the  studies 
is  derived  from  a  myriad  of  factors  when  users  were  asked  to 
evaluate  their  computer  services  relative  to  a  sense  of 
satisfaction  (Bailey  and  Pearson,  1983) .  Factors  measured  in 
the  various  studies  include:  accuracy,  content,  frequency, 
timeliness,  reliability,  assistance,  adequacy,  accommodation, 
communication,  access,  appreciation  and  flexibility.  Each  of 
these  studies  use  a  measure  that  is  unique  to  that  particular 
study  (Ives  et  al .  ,  1983),  but  taken  as  a  whole  provide 
insight  into  a  description  of  UIS. 

Validation  of  the  UIS  measures  in  these  studies  is 
limited,  especially  in  handling  threats  to  internal  validity 
(Haga  and  Zviran,  1990)  .  However,  a  tool  for  measuring  and 
analyzing  UIS  was  developed  by  Bailey  and  Pearson  (1983) , 
based  on  the  previous  studies,  that  provides  the  basis  of  an 
instrument,  later  refined  (Ives  et  al.,  1983),  and  validated 
(Baroudi  and  Orlikowski,  1988)  for  use.  Although  the  method 
employed  to  measure  user  satisfaction  by  this  tool  is  a 
pre-experimental  design  lacking  pretest-postest  or  control 
group  (Campbell  and  Stanley,  1966) ,  Conrath  and  Mignen  (1990) 
claim  it  represents  a  key  contribution  in  the  development  of 
a  standard  instrument  to  measure  user  satisfaction.  The 
development  and  refinement  of  this  instrument  will  be 
described  in  more  detail. 

Bailey  and  Pearson  (1983)  described  a  model  proposed  by 
Lawler  and  Wanous  (1972)  for  measuring  user  satisfaction: 
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where 

Ri;)  =  The  reaction  to  factor  j   by  the  individual  i 
W±j  =  The  importance  of  factor  j   to  individual  i 

Using  this  model,  the  satisfaction  of  the  user  is  measured 
as  the  weighted  sum  of  the  user's  positive  and  negative 
reactions  to  a  set  of  factors  about  the  information  system 
(Bailey  and  Peason,  1983) .  The  user's  perception  of  a  "good" 
information  system  would  be  a  system  that  the  user  is  very 
satisfied  with  the  factors  considered  most  important  in  the 
system. 

To  implement  the  model,  the  set  of  factors  representing 
the  domain  of  user  satisfaction  was  established  via  a  review 
of  22  studies  dealing  with  computer/user  interface  (Bailey  and 
Pearson,  1983) .  The  initial  36  factors  generated  from  the 
literature  review  were  expanded  to  39  factors  after  further 
review  by  middle  managers.  They  concluded  that  the  3  9  factors 
included  represent  the  domain  of  user  satisfaction  at  a=0.01. 

Bailey  and  Pearson  (1983)  used  four  bipolar  adjective 
pairs  on  a  seven  point  Likert-type  scale  to  measure  the  user's 
perception  of  a  factor.  Along  with  the  four  adjective  pairs 
was  included  the  user's  perception  of  importance  of  the 
factor.  The  seven  intervals  from  negative  to  positive  were 
denoted  by  adverbial  qualifiers.   Figure  3.2  represents  their 


16 


use  of  the  semantic  differential  technique  for  measuring  the 
user's  response  to  "Format  of  Output"  factor. 


Format  of  Output:  The  material  design  of  the  layout  and 

display  of  the  output  contents. 


good  : 

simple  : 

readable  : 

useful  : 


To  me,  this  factor  is 
important  : 


:  bad 

:  complex 

:  unreadable 

:  useless 

:  unimportant 


Figure  3.2   Illustration  of  questionnaire  form 

The  Bailey  and  Pearson  user  information  satisfaction 
questionnaire  was  determined  to  be  a  reliable,  valid 
instrument  (Bailey  and  Pearson,  1983) .  It  represented  an 
important  first  step  toward  the  development  of  a  valid  and 
useful  UIS  measure  (Ives  et  al.,  1983). 

Deese  (197  9)  used  the  Pearson  questionnaire  at  the  Federal 
Computer  Performance  Evaluation  and  Simulation  Center.  He 
stated,  "The  results  identified  problems  that  would  not 
otherwise  have  been  discovered"  and  claimed  that  the  user 
satisfaction  questionnaire  was  a  very  useful  and  worthwhile 
tool  (Deese,  1979) . 

Ives,  Olson  and  Baroudi  (1983)  chose  to  undertake  an 
in-depth  assessment  of  the  Pearson  (1983)  questionnaire. 
Replicating  the  initial  Bailey  and  Pearson  study,  they  were 
able  to  reinforce  the  validity  and  reliability  of  the 
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instrument.   They  presented  several  approaches  to  improve  the 
quality  of  the  original  Pearson  instrument. 

The  goal  of  the  suggested  improvements  was  to  establish  a 
standardized  "short  form"  instrument.  Since  Pearson  found  the 
importance  scale  provided  no  additional  information,  it  was 
eliminated  from  the  instrument.  Scales  which  showed 
undesirable  psychometric  qualities  were  also  eliminated.  The 
number  of  items  scaled  within  a  question  was  reduced  from  four 
to  two  in  order  to  reduce  the  time  to  complete  the 
questionnaire.  All  of  the  scales  in  the  original  Pearson 
instrument  were  scored  positively  to  the  left  and  negatively 
to  the  right  end  of  the  scale.  To  reduce  the  undesired 
effects  created  by  a  person  simply  marking  down  a  column  of 
responses,  some  of  the  scales  were  reversed  scored,  thereby 
increasing  the  reliability  of  the  measure.  The  resulting 
short  form  was  determined  to  substantially  measure  the  concept 
of  UIS  originally  proposed  in  the  Pearson  full  instrument 

(Ives  et  al.,  1983) . 

The  short  form  measure  (Ives  et  al . ,  1983)  was  further 
subjected  to  testing  by  Baroudi  and  Orlikowski  (1988)  to 
examine  its  psychometric  properties.  The  final  short  form 
questionnaire  consisted  of  13  questions  with  two  items 
evaluated  per  question  (Appendix  A) .  Each  item  is  scored  on 
a  seven  point  Likert-type  scale  from  -3  (dissatisfied)  to  +3 

(satisfied)  with  zero  indicating  a  neutral  response. 
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The  total  individual  user  information  satisfaction  score 
is  calculated  by  averaging  the  responses  of  the  two  items  for 
each  question  and  summing  the  scores  of  the  13  questions. 


13  R*R 


where 

S±  =  total  individual  user  satisfaction 
F-!  =  response  to  the  first  item  scale 
R2  =  response  to  the  second  item  scale 


The  range  of  total  satisfaction  can  be  from  -39  to  +39. 

Three  factors;  electronic  data  processing  (EDP)  staff  and 
services  (Factor  A)  ,  information  product  (Factor  B)  ,  and 
knowledge  and  involvement  (Factor  C) ;  were  found  to  comprise 
user  satisfaction  using  factor  analysis  by  Ives,  Olson  and 
Baroudi  (1983) .  These  subtotals  are  calculated  as  the  average 
of  the  responses  to  questions  loading  into  a  particular 
factor.  Questions  1,  2,  6,  11,  and  12  load  heavily  into  the 
electronic  data  processing  (EDP)  staff  and  services  factor. 
Questions  7,  8,  9,  10  and  13  load  heavily  into  the  information 
product  factor.  Questions  2,  3,  4,  and  5  load  heavily  into 
the  knowledge  and  involvement  factor.  The  factor  subtotals 
will  range  from  -3  to  +3  in  value.  Averages  are  used  for 
meaningful  comparison  between  the  three  factor  scores . 

Data  were  gathered  for  the  study  from  358  employees, 
mostly  clerical  and  support  personnel,  of  2  6  New  York  area 
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organizations.  Construct  validity  was  determined  first  by 
examining  the  relationship  between  each  scale  and  the  total 
UIS  score.  Factor  analysis  using  varimax  rotation  converged 
to  a  three  factor  solution  accounting  for  68%  of  the  variation 
in  five  iterations.  This  provides  strong  evidence  of 
construct  validity  (Baroudi  and  Orlikowski,  1988) . 

Convergent  validity  was  proven  by  comparing  an  interview 
measure  of  satisfaction  with  the  instrument  measure  in  two 
groups.  One  group  of  users  were  generally  pleased  and 
satisfied  with  the  information  system  and  the  other  group  of 
users  were  generally  dissatisfied  on  interview. 
Administration  of  the  instrument  indicated  a  statistically 
significant  difference  by  t-test  in  the  groups  at  p  <  .001 
(Baroudi  and  Orlikowski,  1988) . 

The  total  satisfaction  and  subtotal  scores  reported  a 
reliability  level  above  the  .80  required  for  research.  This 
demonstrates  the  instrument  is  internally  consistent  and 
reasonably  free  from  measurement  error. 

The  work  by  Baroudi  and  Orlikowski  (1988)  and  resulting 
short-form  UIS  instrument  provides  a  reliable  and  valid 
measure  of  user  information  satisfaction.  It  furnishes  a 
meaningful,  standardized  measure  of  the  overall  satisfaction 
with  an  information  system  as  well  as  specific  information 
about  satisfaction  within  the  subfactors  of  electronic  data 
processing  (EDP)  staff  and  services,  information  product,  and 
user  knowledge  and  involvement. 
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IV.    RESEARCH  METHODOLOGY 

A.  INTRODUCTION 

The  purpose  of  this  research  is  to  identify  the 
characteristics  of  user  information  satisfaction  (UIS)  for 
users  of  the  Composite  Health  Care  System  (CHCS) .  The  study 
is  based  on  statistical  analysis  of  empirical  data  collected 
for  CHCS  users  at  the  Naval  Hospital,  Charleston,  SC.  The 
following  sections  will  discuss  the  survey  instrument,  data 
collection  methodology  and  methods  of  statistical  analysis 
employed  in  the  study. 

B.  THE  SURVEY  INSTRUMENT 

To  avoid  the  pitfalls  associated  with  developing  a 
comprehensive  survey  questionnaire,  the  previously  developed, 
psychometrically  evaluated  and  validated  short-form 
questionnaire  of  Baroudi  and  Orlikowski  (1988)  was  used 
without  alteration.  A  copy  of  the  complete  survey 
questionnaire  is  included  in  Appendix  A.  The  questionnaire  is 
composed  of  two  sections:  a  demographical  section  and  a  user 
information  satisfaction  section. 

1 .   Demographic  Section 

The  first  section  of  the  questionnaire  requested 
general  information.  Questions  1-5,  the  respondents  were 
asked  to  give  their  hospital  department,  job  description, 
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highest  level  of  education,  age,  and  gender  respectively.  To 
determine  the  user's  experience  level  with  the  system,  the 
sixth  question  asked  for  the  length  of  time,  in  months,  the 
respondent  had  used  CHCS .  Further  determination  of  the  user's 
experience  was  sought  in  questions  seven  and  eight.  Question 
seven  asked  if  the  user  had  used  other  computer  systems 
previously.  If  the  respondent  answered  "yes"  to  question 
seven,  he  or  she  was  asked  if  the  previous  system  was  a  health 
care  information  system. 

2 .   User  Information  Satisfaction  Section 

The  second  part  of  the  survey  directly  addressed  the 
issues  of  user  information  satisfaction.  The  instrument  was 
designed  to  elicit  the  user's  perception  of  how  well  the  CHCS 
system  functioned  in  their  area.  It  consisted  of  thirteen 
questions;  each  question  having  two  bipolar  adjective  Likert- 
type  item  scales  for  responses.  Each  item  can  take  on  one  of 
seven  values  from  -3  (extremely  dissatisfied)  to  +3  (extremely 
satisfied)  with  zero  indicating  a  neutral  response. 

The  questions  asked  can  be  categorized  into  one  of 
three  factors  of  user  satisfaction. 

Management  Information  Services  (MIS)  Staff  and 
Services :  This  factor  is  the  respondent's  perception  of  the 
attitude  and  responsiveness  of  the  MIS  staff  and  the 
relationship  with  the  MIS  staff.  Due  to  the  fact  that  the  MIS 
personnel  at  the  Naval  Hospital,  Charleston  provide  only  local 
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support  and  act  as  liaison  to  the  contractor  that  supplies  the 
software  product,  this  factor  is  further  broken  down  to 
reflect  services  provided  by  the  local  MIS  department  and  the 
contractor. 

Information  Product:  This  factor  is  the  respondent's 
perception  of  the  quality  of  output  delivered  by  CHCS. 

Knowledge  and  Involvement ;  This  factor  is  the 
respondent's  perception  of  the  quality  of  training  provided, 
their  understanding  of  the  system,  and  their  participation  in 
its  development . 

C.   SAMPLE  AND  DATA  COLLECTION 

Naval  Hospital,  Charleston  is  one  of  thirteen  beta-test 
sites  for  CHCS  software.  It  was  selected  as  the  site  to 
conduct  the  survey  because  all  functional  modules  were  to  be 
operational  at  the  time  of  the  survey.  Since  the  in-patient 
modules,  servicing  nursing  and  clinical  dietetics,  were  not 
implemented  at  the  time  of  data  collection,  the  sample 
population  then  consisted  of  personnel  from  out-patient  and 
administration  areas. 

A  local  point  of  contact  acted  as  a  distribution  and 
collection  agent  for  the  surveys.  The  questionnaires,  each 
accompanied  with  a  cover  letter  (Appendix  B) ,  were  mailed  to 
the  point  of  contact  at  the  site.  The  point  of  contact 
distributed  180  questionnaires  and  return  envelopes  to  all 
department  personnel  within  the  hospital  operating  the  CHCS 
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modules .  The  surveys  were  completed  and  returned  by  the 
respondents  to  the  point  of  contact  in  sealed  envelopes  and 
then  the  questionnaires  were  forwarded  to  the  researcher  for 
analysis . 

110  questionnaires  were  returned,  of  which  101  surveys  had 
complete  information.  This  represents  a  final  return  rate  of 
56%. 

D.   STATISTICAL  ANALYSIS  METHODS 

Data  from  completed  surveys  were  coded  and  entered  into  a 
spreadsheet  program.  The  spreadsheet  program  was  used  to 
perform  the  statistical  analysis.  The  specific  procedures 
used  in  the  analysis  will  be  presented  next. 

1.   Respondent's  Satisfaction  Index 

Each   respondent's   total   satisfaction   index   was 
calculated  for  the  13  questions  using  the  following  formula: 

13  R      +R 


where: 

S±    =  Satisfaction  Index  for  respondent  i 

Raij  =  Response  to  first  item  scale  of  question  j  for 

repondent  i 
R^  =  Response  to  second  item  scale  of  question  j  for 

respondent  i 
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2 .  Total  Satisfaction  Index 

The  overall  or  total  satisfaction  index  for  the  survey 
group  is  calculated  by  averaging  the  respondent's  satisfaction 
index  to  find  the  mean. 

3 .  Question  Averages 

The  mean  response  to  each  of  the  thirteen  questions  is 
calculated  for  comparison  of  individual  questions. 

4 .  Factor  Subtotals 

The  thirteen  questions  can  be  grouped  into  three 
factor  subtotals  as  was  defined  by  Ives,  Olson,  and  Baroudi 
(1983)  :  MIS  staff  and  services  (Factor  A)  ;  Information  product 
(Factor  B)  ;  and  Knowledge  and  involvement  (Factor  C)  .  The 
mean  of  the  individual  questions  averages  is  calculated  to 
find  the  factor  subtotals.  Questions  1,  2,  6,  11,  and  12  are 
used  for  the  MIS  staff  and  services  subtotal.  Questions  7,  8, 
9,  10,  and  13  are  used  for  the  information  product  subtotal. 
Questions  2,  3,  4,  and  5  are  used  for  the  knowledge  and 
involvement  subtotal. 

The  MIS  staff  and  services  sub-factor  is  further 
subdivided  to  provide  insight  into  the  distinction  between  the 
local  Management  Information  Department  (MID)  department 
functions  and  the  functions  of  the  software  contractor. 
Again,  the  mean  of  the  individual  question  averages  is 
calculated  to  subdivide  the  MIS  staff  and  services  factor. 
Questions  1,  6,  and  11  directly  address  the  attitude,  services 
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and  relationship  with  the  local  MID  are  used  for  the  local  MID 
factor.   Questions  2  and  12  address  issues  associated  with 
software  developement,  the  responsibility  of  the  civilian 
contractor,  and  are  used  for  the  contractor  factor. 
5 .   Comparison  of  Groups 

The  respondents  were  categorized  by  the  type  of  use  of 
the  CHCS  system.  Three  work  groups  were  identified:  (1) 
Physicians;  (2)  Ancillary;  and  (3)  Administration.  The 
individual  question  averages,  total  satisfaction  indexes,  and 
sub-factor  totals  were  compared  for  one  group  verses  the 
combination  of  the  other  two  groups  using  a  t-test  with  the 
significance  level  of  alpha  =  0.05.  The  t-test  is  used  to 
determine  if  there  is  a  significant  difference  between  the 
arithmetic  mean  value  of  two  groups. 
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DESCRIPTIVE  FINDINGS 


A.   DEMOGRAPHIC  FINDINGS 

Of  the  101  respondents,  57  were  male  and  44  were  female. 
The  age  of  the  respondents  ranged  from  19  to  56  with  a  mean  of 
32  years  old. 

1 .   Work  Center  Distribution 

The  hospital  department  work  centers  reported  were 
out-patient  areas  including:  administration,  clinics,  pharmacy 
and  laboratory  departments.  The  CHCS  modules  for  in-patient 
areas  were  not  operational  at  the  time  of  the  survey. 
Personnel  from  in-patient  areas  were  not  included  in  the 
study.   Figure  5.1  reflects  the  distribution  by  work  center. 


DEPARTMENT  WORK  CENTERS 


CLINICS 


LABORATORY 
18 


ADMINISTRATION 

g 


Figure   5 . 1 
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2 .   Job  Description  Distribution 

The  job  descriptions  reported  are  from  a  multitude  of 
specific  functions  but  can  be  categorized  as: 

1 .  Technician  -  a  hospital  corpsman  functioning  in  a 
medical-technical  capacity  (pharmacy  technician, 
laboratory  technician,  clinic  corpsman) 

2 .  Physician  -  a  medical  doctor 

3.  Health  Professional  -  a  licensed  medical 
professional  other  than  a  physician  (nurse, 
pharmacist,  physical  therapist,  bio-medical 
officer) 

4.  Other  -  a  person  not  listed  above  (hospital 
administration  non-medical  clinic  staff) 

Figure  5.2  reflects  the  distribution  by  job  description. 


JOB  DESCRIPTION 


TECHNICIAN 
61 


PHYSICIAN 
21 


HEALTH  PROFESSIONAL 
12 


Figure   5 . 2 
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3.   Work  Group  Distribution 

For  the  purposes  of  this  investigation,  the  study 
population  was  segregated  according  to  the  assumed  primary  use 
of  the  computer  system.  The  categories  provide  a  clear 
separation  of  work  groups  from  an  organizational  and  cultural 
standpoint.  These  groups  are  used  extensively  throughout  the 
study . 

1 .  Administrative  -  use  of  the  system  for  indirect 
medical  purposes  (patient  registration, 
appointment  scheduling  and  administrative 
reporting) 

2.  Ancillary  -  use  of  the  system  for  direct  medical 
support  (prescription  filling,  laboratory  specimen 
processing) 

3.  Physician  -  use  of  the  system  by  physicians 

(accessing  patient  data,  transmission  of  orders) 
Figure  5.3  represents  the  population  of  respondents  according 
to  their  main  use  of  the  system. 
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Figure  5 . 3 


4 .   Education  Distribution 

The  level  of  education  reported  by  the  respondents 
represents  a  well  educated  user  population.  Almost  half  of 
the  users  have  a  college  degree  and  almost  ninety  percent  have 
had  some  college  education.  Figure  5.4  represents  the 
breakdown  by  education. 
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Figure   5 . 4 


B.  COMPUTER  SYSTEM  USE 

The  length  of  time  the  respondents  reported  as  having  used 
the  CHCS  computer  system  ranged  from  one  month  to  18  months 
with  a  mean  of  8.7  months.  Of  the  101  respondents,  83  had 
used  a  computer  system  previously.  Only  24  of  the  83  users 
had  used  a  health  care  information  system  before  the  CHCS 
implementation.  Although  many  of  the  respondents  had 
experience  with  computers,  this  was  the  first  exposure  to  a 
health  care  information  system  for  7  6  percent  of  the  users. 

C.  SATISFACTION  FINDINGS 

The  survey  results  for  user  satisfaction  are  compared  for 
the  three  work  groups  in  three  areas:  1)  overall  satisfaction, 
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2)  responses  to  individual  questions,  3)  three  group  factors: 
MIS  staff  and  services,  information  product,  and  knowledge  and 
involvement .  The  actual  values  reported  in  the  survey 
responses  are  not  significant  in  themselves,  rather  the 
relative  scores  among  individual  questions,  sub- factors  and 
work  groups  are  useful  for  comparison.  Since  ancillary 
composes  46%  of  the  survey  group,  the  survey  averages  will 
tend  to  more  closely  reflect  the  responses  of  the  ancillary 
group  than  the  other  groups.  Therefore,  average  scores  on 
individual  questions  are  not  fully  representative  and  are 
included  only  for  rough  comparisons. 
1 .   Overall  Satisfaction 

Overall  satisfaction  is  measured  by  the  mean  of  the 
sum  of  the  responses  to  the  survey  questions.  The  mean  survey 
score  index  was  11.26  on  a  scale  ranging  from  -39  to  +39. 
Figure  5.5  presents  a  comparison  of  the  overall  satisfaction 
index  the  three  groups  and  the  overall  mean  satisfaction.  The 
survey  shows  that  the  administrative  group  displays  the 
highest  overall  satisfaction  (14.96),  followed  by  ancillary 
(9.65),  with  physicians  recording  the  least  overall 
satisfaction  (8.07) .  Using  a  t-test,  there  is  no  significant 
difference  between  any  one  work  group  and  the  remaining  groups 
at  alpha  =  0.05. 
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Figure  5 . 5 


2 .   Individual  Questions 

Each  of  the  13  questions  on  the  survey  has  two 
responses  from  the  user.  The  response  scales  were  reversed 
scored  on  many  of  the  questions  to  neutralize  the  effect  of  an 
individual  marking  boxes  straight  down  the  questionnaire. 
Figure  5.6  shows  the  average  level  of  satisfaction  for 
individual  questions  by  work  group. 
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Figure  5 . 6 


It  is  readily  apparent  that  some  questions  exhibit  a 
relatively  high  or  low  level  of  satisfaction  when  compared  to 
the  other  questions.  Three  questions  (question  1,  6  and  11) 
stand  out  from  the  others  for  the  high  level  of  satisfaction 
expressed  by  the  users.  Each  of  these  questions  deal  with 
issues  concerning  the  Management  Information  Services  (MIS) 
staff.  Two  questions  stand  out  as  exhibiting  an  extremely  low 
(negative)  satisfaction  index.  Questions  2  and  12  are 
concerned  with  software  development  time.  A  complete 
decomposition  of  each  question  by  work  group  will  follow. 

Figure  5.7  displays  the  responses  to  individual 
questions  by  physicians  compared  to  the  remainder  of  the 
respondents;  Figure  5.8  administrative  compared  to  the 
remainder;  Figure  5.9  ancillary  compared  to  the  remainder. 
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Figure  5 . 9 

a.  Question     1     -    Relationship     with     the    Management 
Information  Department    (MID)    Staff 

This  question  measures  the  level  of  constructive 
interaction  between  the  user  and  the  Management  Information 
Department.  As  noted  earlier,  the  level  of  satisfaction 
towards  the  MID  staff  is  relatively  high.  Although  not 
statistically  significant  at  alpha  =  0.05,  the  administrative 
group  showed  the  highest  level  of  satisfaction  of  the  three 
groups.  Slightly  lower  was  the  ancillary  group  followed  by 
the  physicians. 

b.  Question  2  -  Processing  of  Requests  for  Changes  to 
Existing  Systems 

The  requested  responses  were  concerned  with  the 
length  of  time  required  and  timeliness  of  changes  to  the  CHCS 
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system.  The  overwhelmingly  negative  level  of  satisfaction; 
the  lowest  scoring  question;  indicates  a  perceived  problem  by 
the  users.  Ancillary  users  were  the  least  satisfied, 
physicians  slightly  higher  and  administrative  users,  albeit 
still  negative,  the  highest  group.  A  t-test  indicated  no 
significant  differences  at  alpha  =  0.05. 

c.  Question  3  -  Degree  of  Training  Provided  to  Users 
This  question  asked  if  the  users  were  completely 

trained  to  use  the  system.  The  level  of  satisfaction  of  the 
administrative  group  was  statistically  higher  at  alpha  =0.05 
than  the  other  groups  combined.  Correspondingly,  the 
ancillary  group  was  significantly  less  satisfied  with  the 
training.  The  physicians  had  the  lowest  mean  score  but  did 
not  prove  to  be  statistically  different. 

d.  Question   4   -   User's   Understanding  of  the   System 
The  level  at  which  a  users  understands  a  system 

naturally  flows  from  training  and  experience.  As  expected, 
the  response  to  this  question  closely  parallels  the  users 
perception  of  the  degree  of  training  provided.  In  this  case, 
physicians  were  significantly  less  satisfied  at  alpha  =  0.05. 
Of  particular  note  is  the  significant  improvement  in 
satisfaction  in  the  ancillary  group  from  their  satisfaction  of 
training. 
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e.  Question  5  -  User's  Feeling  of  Participation 

The  user's  perception  of  participation  is 
beneficial  for  the  success  of  a  computer  system.  Lack  of 
participation  may  lead  to  dissatisfaction  with  the  system.  In 
response  to  this  question,  the  administrative  group  had  a 
statistically  significant  higher  feeling  of  participation  than 
the  other  groups.  Physicians  perceived  the  least  amount  of 
participation.  Ancillary  personnel  responded  more  positively 
than  physicians  but  less  positively  than  administrative. 

f.  Question  6  -  Attitude  of  the  Management  Information 
Department    (MID)    Staff 

The  willingness  and  commitment  of  MID  staff  to  lend 
assistance  for  the  benefit  of  the  users  is  measured  by  their 
perceived  attitude  towards  the  users.  The  overall  response  to 
this  question  (the  highest  rated  question)  indicates  that  MID 
displays  a  highly  cooperative  and  positive  attitude.  All 
three  groups  responded  favorably.  However,  administration 
reported  a  statistically  significant  (alpha  =  0.05)  higher 
response  than  the  other  groups.  Although  the  physicians 
scored  this  question  the  highest  of  all  13  questions,  their 
response  was  statistically  lower  than  the  other  groups. 

g.  Question    7  -  Reliability  of  Output   Information 
Users  must  be  able  to  accept  the  output  information 

provided  by  the  computer  system  as   reliable.     This   is 
extremely  important  for  medical  information  systems.    The 
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responses  were  positive  with  no  significant  differences  among 
the  groups.  Of  note  is  the  fact  that  physicians  responded 
more  favorably  than  the  other  groups  and  ancillary  the  least 
favorably. 

h.    Question    8    -  Relevancy  of  Output    Information    (to 
Intended  function) 

This  question  measures  the  degree  of  congruence 
between  what  the  user  wants  or  requires  and  what  is  provided 
by  the  information  products  and  services.  All  groups 
responded  positively  without  statistically  significant 
differences.  The  administrative  group  was  most  satisfied  with 
the  relevancy  of  output  of  the  three  groups. 

1 .    Question   9   -  Accuracy  of  Output   Information 

All  three  groups  were  satisfied  that  the  system  was 
providing  correct  information.  Of  the  three  groups,  the 
physicians  were  the  most  satisfied  with  the  accuracy  of  output 
information.  Ancillary  and  administrative  groups  were  about 
equally  satisfied.  The  differences  in  responses  were  not 
statistically  significant. 

j.  Question   10   -  Precision   of  Output   Information 

Precision  of  output  information  is  the  variability 
of  output  information  from  that  which  it  purports  to  measure. 
The  response  was  favorable,  but  was  the  lowest  of  the 
questions  answered  positively.  The  ancillary  group  responded 
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statistically  lower  than  the  other  groups  on  this  question 
while  the  physicians  recorded  the  highest  score. 

k.    Question    11    -    Communication    with    the    Management 

Information  Department    (MID)    Staff 

Communication  is  the  manner  and  methods  of 
information  exchange  between  the  users  and  the  MID  staff.  As 
noted  previously,  all  questions  concerning  MID  staff  received 
high  satisfaction  levels  from  all  the  groups.  None  of  the 
groups'  responses  were  statistically  different  by  t-test  at 
alpha  =  0.05. 

1 .    Question     12     -     Time     Required     for     New     Systems 

Devel opement 

New  systems  development  time  is  the  time  between 
user-initiated  request  for  service  or  action  and  a  reply  to 
that  request.  Much  like  the  processing  of  requests  for 
changes  to  existing  systems,  the  groups  were  either  neutral  or 
highly  dissatisfied.  The  physicians  were  statistically  less 
satisfied  than  the  other  groups;  reporting  the  lowest  index  of 
any  question. 

m.    Question   13   -  Completeness   of  Output 

All  three  groups  were  adequately  satisfied  with  the 
comprehensiveness  of  the  content  of  information  product.  The 
level  of  satisfaction  of  each  group  was  similar  with  the 
administrative  group  the  most  satisfied. 
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3.   Grouped  Factors 

The  questions  are  grouped  into  three  factors  that 
comprise  user  satisfaction  described  by  Ives,  Olsen,  and 
Baroudi  (1983).  These  factors:  MID  staff  and  services, 
information  product,  and  knowledge  and  involvement,  are 
calculated  through  averages  of  the  component  questions.  MID 
staff  and  services  (Factor  A)  is  sub-divided  into  two  separate 
components:  local  MID  staff  functions  (MID)  and  contractor 
functions  (Contractor)  .  Factor  A  (MID)  is  derived  from 
questions  1,  6, and  11;  Factor  A  (Contractor)  from  questions  2 
and  12;  information  product  (Factor  B)  from  questions  7,  8,  9, 
10  and  13;  and  knowledge  and  involvement  (Factor  C)  from 
questions  2,  3,  4,  and  5. 

None  of  the  work  groups'  satisfaction  level  to  any  of 
the  factors  were  statistically  different  at  significance  level 
alpha  =  0.05.  However,  general  observations  can  be  made. 
Figure  5.10  represents  the  work  groups'  level  of  satisfaction 
with  relation  to  sub-total  factors. 
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Figure  5.10 


Figure   5.11   represents   factor   satisfaction   of 
physicians  compared  to  other  respondents. 
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42 


Figure   5.12   represents   factor   satisfaction   of 
administrative  personnel  compared  to  other  respondents. 
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Figure   5.13   represents   factor   satisfaction   of 
ancillary  personnel  compared  to  other  respondents. 
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Figure  5.13 


a.  Factor  A    (Local  MID  Staff  and  Services) 

This  factor  represents  the  attitude,  services  and 
relationship  with  the  local  MID  personnel.  As  seen  in  Figure 
5.10,  the  administrative  personnel  displayed  the  highest  level 
of  satisfaction.  The  lowest  level  of  satisfaction,  although 
still  extremely  positive,  was  displayed  by  the  physicians. 
Jb.  Factor  A    (Contractor  Services) 

This  factor  represents  the  responsiveness  and 
services  of  software  programmers.  This  factor  received  the 
lowest  level  of  satisfaction  of  any  factor.  All  work  groups 
rated  this  factor  negatively,  with  ancillary  personnel 
recording  the  lowest  level  of  satisfaction  of  any  group. 
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c.  Factor  B  -  Information  Product 

This  factor  represents  the  users  perception  of  the 
quality  of  output  from  the  system.  Although  the  differences 
among  the  groups  were  not  statistically  significant ,  the 
administrative  group  scored  the  highest  level  of  satisfaction. 
Ancillary  members  presented  the  lowest  level  of  satisfaction. 

d.  Factor  C  -  Knowledge  and  Involvement 

This  factor  represents  the  respondents'  self- 
reported  assessment  of  the  quality  of  training  provided, 
their  understanding  of  the  system,  and  their  participation  in 
its  development.  There  was  a  statistically  significant 
difference  between  the  level  of  satisfaction  of  the 
administrative  group  and  the  other  respondents. 
Administrative  personnel  were  highly  satisfied  with  their 
knowledge  and  involvement  of  the  system. 
4 .   Time  of  System  Use 

As  users  become  familiar  with  the  functions  of  a 
system  over  time,  it  is  expected  that  the  more  experienced 
users  would  exhibit  a  greater  level  of  satisfaction. 

Separating  the  respondents  into  subgroups  of  users 
with  experience  less  than  six  months,  six  months  to  11  months, 
and  greater  than  11  months  provides  subgroups  of  approximately 
equal  size.  The  work  groups  exhibit  roughly  the  same 
proportions.  Figure  5.14  represents  the  breakdown  of 
respondents  time  of  system  use. 
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Figure  5 . 14 


All  the  respondents,  broken  down  by  time  of  system 
use,  demonstrate  that  user  satisfaction  did  not  appreciably 
change  with  the  increase  in  experience  of  users.  Figure  5.15 
shows  the  level  of  satisfaction  with  CHCS  verses  time  of 
system  use. 
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Figure  5 . 15 


None  of  the  work  groups;  administrative,  physicians, 
or  ancillary;  exhibited  any  high  correlation  between  time  of 
system  use  and  level  of  satisfaction  of  any  factor  by 
regression  analysis.  However,  it  is  noteworthy  to  report  the 
trends  in  the  level  of  satisfaction  for  the  work  groups  over 
time. 

The  administrative  work  group  respondents  with  less 
than  six  months  system  use  report  a  satisfaction  index  of 
8.71.  As  the  users  gain  experience  with  the  system  of  six 
months,  the  level  of  satisfaction  increases  to  17.42. 
Satisfaction  remains  consistent,  at  17.29,  for  users  with 
greater  than  11  months  experience.  Figure  5.16  shows  the 
relationship  of  the  satisfaction  of  administrative  users  to 
time  of  system  use. 
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On  the  other  hand,  physicians'  level  of  satisfaction 
is  consistent  (satisfaction  index  6.50-6.37)  for  users  up  to 
11  months  of  experience.  The  level  of  satisfaction  increases 
for  users  with  greater  than  11  months  of  experience.  Figure 
5.17  shows  physician  satisfaction  verses  time  of  system  use. 
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Figure   5 . 17 


Unlike  either  the  administrative  or  physician  work 
groups,  ancillary  respondents  with  the  least  experience  in 
system  use  demonstrated  the  greatest  satisfaction 
(satisfaction  index  16.90).  The  level  of  satisfaction 
declined  to  11.44  for  users  with  six  to  11  months  experience. 
Satisfaction  further  declined  to  1.56  for  users  with  greater 
than  11  months  experience.  Figure  5.18  shows  the  level  of 
satisfaction  verses  time  for  ancillary  users. 


49 


Ancillary 

Satisfaction  verses  Time  of  Use 


20.00 


<D   16.00 

C 
C 

5 

'■g  10.00 
CO 

To 

I 

W     6.00 


0.00 


More  than 
8  Months 


16.00 

11.44 

} 

| i 

1 ! + 

»                             1.66 

:Ji             B8888888888 

6-11 
Months 

Tim*  or  System  uta 


More  than 
11  Months 


Figure   5.18 
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VI.   ANALYSIS  OF  FINDINGS 

A.   INTRODUCTION 

The  purpose  of  this  research  is  to  describe  and  document 
the  user  information  satisfaction  characteristics  of  the  users 
of  the  Composite  Health  Care  System  at  the  Naval  Hospital, 
Charleston.  The  study  results  are  not  to  be  seen  as  a 
definitive  evaluation  of  CHCS,  but  provide  a  baseline 
reference  point  for  identifying  system  strengths  or  possible 
areas  of  conflict  and  dissatisfaction.  Areas  identified  as 
dissatisfying  should  be  examined  further  through  interviews 
and  by  examining  the  development  and  operating  procedures  of 
the  work  group  to  determine  the  possible  causes  and  course  of 
action. 

The  value  of  the  satisfaction  scores  (positive  or 
negative)  is  an  important  finding  in  itself,  however,  the 
relative  comparison  of  scores  across  different  groups  of  users 
is  useful  for  the  purposes  of  this  study. 

The  analysis  of  the  survey  findings  will  primarily  focus 
on  the  differences  and  similarities  among  the  three  work 
groups:  administrative,  ancillary,  and  physicians .  The  first 
section  will  look  at  the  overall  user  information  satisfaction 
with  CHCS.  The  second  section  will  discuss  significant 
differences  in  individual  question  responses  reported  between 
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work  groups.  The  third  section  will  compare  the  three  work 
groups  with  respect  to  the  grouped  factors:  local  MIS  staff 
and  services,  contractor  services,  knowledge  and  involvement, 
and  information  product.  The  last  section  will  look  at  user 
satisfaction  for  the  three  work  groups  from  the  perspective  of 
time  of  system  use. 

B.   OVERALL  SATISFACTION 

1 .  Combined  Groups 

The  overall  satisfaction  index  is  a  general 
description  of  the  user's  perception  of  CHCS  as  a  whole.  A 
negative  result  index  would  indicate  dissatisfaction  while 
positive  results  index  some  degree  of  satisfaction  with  the 
system.  The  survey  average  satisfaction  index  reported  is 
11.26.  Since  the  overall  satisfaction  index  ranges  from  -39 
to  39  in  scale,  the  respondents  average  overall  satisfaction 
is  characterized  as  slightly  satisfied  using  the  adjective 
qualifiers  of  Baroudi  and  Orlikowski  (1988) . 

2 .  Comparison  of  Work  Groups 

The  three  work  groups  used  in  this  study  were  chosen 
to  differentiate  the  users  by  the  way  they  primarily  use  the 
computer  system.  There  were  no  significant  differences  found 
in  overall  satisfaction  between  any  one  work  group  and  the 
remaining  groups.  Although  all  groups  are  considered  slightly 
satisfied  with  CHCS,  there  are  meaningful  differences  between 
the  work  groups'  overall  satisfaction. 
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Lowest  of  the  work  groups  in  overall  satisfaction  was 
the  physician  group  (8.07).  Physicians  use  CHCS  almost 
exclusively  for  the  conduct  of  direct  patient  care.  The 
functionality  of  the  system  for  physicians  is  the  most  diverse 
of  the  three  groups.  Requirements  of  computer  input  are 
different  for  medical  record  queries ,  prescribing  medications, 
laboratory  tests,  radiology  requests,  and  so  on.  The 
physician  needs  to  know  the  procedures  for  each  type  of 
functionality  to  use  the  system  effectively. 

Each  physician  has  a  terminal  located  in  the  office. 
When  using  the  system  for  out-patient  purposes,  the  patient 
frequently  is  present  in  the  office  at  the  time  of  use.  In 
addition,  the  medical  profession  have  not  traditionally  used 
automated  means  to  document  patient  treatments  or  write 
prescriptions  in  the  past.  The  computer  system  can  be 
considered,  by  them,  an  intrusion  into  their  already  busy 
schedule  and  has  caused  the  physicians  to  alter  their  work 
patterns  to  accommodate  the  system. 

The  significant  change  of  work  habits,  extensive 
functionality  requirements,  time  constraints  imposed  on  the 
physicians,  and  the  presence  of  patients  in  the  office  while 
operating  the  system  combine  to  make  this  finding  not 
surprising. 

Ancillary  personnel,  the  largest  of  the  work  groups 
(46%) ,  demonstrated  a  slightly  higher  overall  satisfaction 
score  (9.65)  .    Most  of  the  users  in  this  work  group  are 
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technicians  using  the  system  to  process  prescriptions, 
laboratory  specimens,  or  radiology  requests.  Ancillary  areas 
in  the  hospital  have  had  automated  processing  capabilities,  in 
some  form  or  another,  for  many  years  and  are  very  accustomed 
to  general  computer  use.  Due  to  the  large  number  of  items 
processed  each  day  in  these  areas,  the  users  are  under  a  tight 
time  constraint  to  process  items  through  the  computer  quickly. 
The  administrative  group  scored  the  highest  overall 
satisfaction  scores.  Administrative  personnel  use  the  system 
for  purposes  other  than  for  the  direct  conduct  of  patient 
care.  With  a  few  exceptions,  the  work  performed  on  the  system 
can  be  accomplished  without  the  pressure  of  a  limited  and 
specific  time  constraint.  The  computer  system  can  therefore 
be  used  at  a  comparatively  slower  pace  than  the  other  work 
groups.  Once  again  with  a  few  exceptions,  most  of  the  work 
performed  by  the  administrative  personnel  previous  to  CHCS  was 
performed  manually.  Automating  the  compilation  and  generation 
of  reports  can  significantly  reduce  the  time  spent  on  these 
activities . 

C.   SIGNIFICANT  INDIVIDUAL  QUESTION  DIFFERENCES 

Although  there  are  demonstrated  differences  in 
satisfaction  among  the  various  groups  in  every  question,  only 
the  questions  with  significant  differences  among  the  work 
groups  will  be  discussed  in  this  section.   The  individual 
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questions   will   be   combined   into   the   grouped   factors 
representing  satisfaction  in  the  next  section. 

1 .  Degree  of  Training  Provided  to  Users 

The  results  of  the  survey  show  the  physicians  were  the 
least  satisfied  with  the  training  provided,  followed  closely 
by  the  ancillary  personnel.  There  is  a  significant  increase 
in  the  satisfaction  with  user  training  expressed  by  the 
administrative  personnel. 

Effective  user  training  is  extremely  important  to  the 
success  of  any  computer  system.  The  person's  satisfaction 
with  the  initial  user  training  received  sets  the  stage  for  his 
or  her  satisfaction  while  operating  the  system.  Often,  user 
training  is  intensive  only  during  the  time  a  system  is  being 
implemented.  After  implementation  new  users  are  frequently 
required  to  learn  the  functionality  of  a  computer  system  with 
On-the- job-training  (OJT) .  Formalized  training  programs  are 
difficult  to  conduct  on  a  regular  basis  due  to  the  constant 
change-over  of  personnel  experienced  at  medical  treatment 
facilities  and  the  specialized  requirements  of  the  different 
work  groups.  Satisfaction  with  the  degree  of  training 
provided  is  closely  associated  with  the  user' s  understanding 
of  the  system  and  their  feeling  of  participation. 

2.  User's  Understanding  of  the  System 

Users  generally  attain  a  high  level  of  understanding 
of  a  computer  system  from  either  effective  training  or 
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experience  or  both.  As  anticipated,  the  work  groups  that 
reported  a  low  level  of  satisfaction  in  user  training 
(Physicians  in  particular)  demonstrated  a  significantly  lower 
level  of  satisfaction  with  their  understanding  of  the  system. 
The  diverse  functionality  of  the  module  used  by  physicians  may 
also  diminish  their  perception  of  the  level  of  understanding. 
3.   User's  Feeling  of  Participation 

The  user's  perception  of  the  shared  involvement  and 
commitment  to  the  operation  of  the  computer  system  is 
beneficial  to  the  success  of  a  computer  system.  Satisfaction 
with  the  level  of  participation  naturally  follows  from  the 
satisfaction  of  user  training  and  the  relationship  with  the 
MID  staff.  It  is  not  surprising  to  observe  the  administrative 
work  group  scoring  significantly  higher  than  the  other  groups 
on  this  question. 

D.   GROUPED  FACTORS 

Baroudi  and  Orlikowski  (1988)  argue  that  the  individual 
questions  can  be  grouped  into  three  factors,  MIS  staff  and 
services,  information  product,  and  knowledge  and  involvement, 
that  comprise  user  satisfaction.  As  discussed  in  Chapter  V, 
the  functions  normally  provided  by  a  MIS  of  the  companies 
studied  in  their  research  are  provided  by  two  separate 
entities  at  Naval  Hospital,  Charleston.  Local  support 
services  are  provided  by  the  Command' s  Management  Information 
Department,   whereas   software   support   is   furnished  by  a 
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civilian  software  contractor.  It  is  logical  to  divide  the 
single  factor,  MIS  staff  and  services  found  in  the  original 
study,  to  make  two  separate  factors:  local  MID  staff  and 
services  and  Contractor's  services. 

The  three  work  groups  generally  scored  similarly  in  each 
of  the  four  factors.  The  only  exception  being  the 
statistically  significant  high  level  of  satisfaction  of  the 
administrative  group  for  the  knowledge  and  involvement  factor. 

The  most  notable  finding  when  four  factors  are  examined  is 
the  significant  negative  level  of  satisfaction  of  all  the  work 
groups  with  the  services  provided  by  the  software  contractor. 
The  services  specifically  asked  in  the  questionnaire  dealt 
with  satisfaction  of  the  time  for  system  changes  and  new 
system  development.  The  negative  satisfaction  finding  for 
this  factor  is,  however,  expected.  Bailey  and  Pearson  (1983) 
list  these  two  factors  as  the  two  most  frequent  for  causing 
dissatisfaction  among  users;  a  finding  supported  by  other 
published  reports. 

At  the  other  end  of  the  spectrum,  the  user's  perception  of 
the  local  MID  staff  and  services  is  considered  quite 
satisfactory.  There  does  not  appear  to  be  any  communication 
or  attitudinal  barriers  to  prevent  an  interchange  of  ideas 
between  the  respondents  and  the  local  MID  staff. 

Lower  in  satisfaction  is  the  user's  perception  of  the 
information  product  factor.  The  information  product  measures 
the  quality  of  the  information  output  provided  to  the  user. 
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The  three  work  groups  combined  rated  this  factor  as  slightly 
satisfactory.  The  users  perceive  the  system  positively  in 
regards  to  the  information's  accuracy,  variance,  reliability, 
and  relevancy  to  the  intended  function. 

The  physician  and  ancillary  work  groups  displayed  a 
positive  satisfaction  index  for  the  knowledge  and  involvement 
factor.  However,  the  satisfaction  level  demonstrated,  albeit 
positive,  was  very  close  to  neutrality.  Administrative 
personnel  indicated  a  satisfaction  level  significantly  higher 
than  the  other  two  groups;  high  enough  to  be  considered 
slightly  satisfied  with  the  factor.  The  combination  of 
satisfaction  with  user  training,  system  understanding,  and 
participation  is  responsible  for  administrative  work  group' s 
higher  level  of  satisfaction.. 

E.   SATISFACTION  VERSES  TIME  OF  SYSTEM  USE 

Satisfaction  with  a  computer  system  is  expected  to  improve 
the  longer  users  operate  a  system  over  time;  gaining  more 
experience  and  understanding  of  the  system. 

An  increase  in  overall  satisfaction  resulting  from  longer 
time  of  system  use  did  not  emerge  from  this  study.  The  study 
did  show,  when  considering  the  survey  population  as  a  whole, 
that  user' s  satisfaction  remained  constant  when  compared  to 
length  of  time  the  respondent  had  used  the  system. 

As  discussed  in  Chapter  V,  the  Ancillary  group  making  up 
46%   of   the   survey   population   tends   to   influence   all 
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generalizations  drawn  from  total  survey  population  findings. 
Breaking  down  the  findings  by  work  groups  demonstrates  this 
fact  well. 

Positive  trends  in  satisfaction  for  the  administrative  and 
physician  work  groups  were  displayed.  Personnel  in  those  work 
groups  who  have  used  the  system  the  longest  (greater  than  11 
months)  were  more  satisfied  with  CHCS  than  those  who  had  used 
the  system  less  than  six  months. 

The  work  performed  by  CHCS  in  the  physician  and 
administrative  work  groups  was  previously  performed  manually. 
As  they  worked  with  the  system,  their  experience  and 
confidence  with  CHCS  expanded,  thereby  increased  satisfaction 
was  a  result. 

Administrative  personnel,  satisfied  with  the  training 
provided,  quickly  grasped  the  system  procedures .  The 
structured  application  of  CHCS  to  their  work  performed 
assisted  in  their  acceptance  of  the  system.  Rapid  increases 
in  satisfaction  with  CHCS  resulted. 

The  complex  functionality  of  the  physician  modules  is 
possibly  responsible  for  their  relatively  prolonged  time 
period  before  satisfaction  is  increased.  After  physicians 
learned  the  multitude  of  inputs  necessary,  their  confidence 
and  satisfaction  with  CHCS  improved. 

Ancillary  personnel  findings  indicate  a  steady  decrease  in 
satisfaction  with  CHCS  for  those  who  have  used  the  system  the 
longest  when  compared  to  those  persons  having  less  than  six 
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months  exposure  to  system  use.  This  is  the  opposite  to  what 
was  anticipated.  It  is  not  clear  why  this  finding  occurred. 
An  in-depth  analysis  of  their  policies  and  procedure  along 
with  interviews  with  the  personnel  is  necessary  to  determine 
the  cause  of  this  finding. 
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VII.   CONCLUSIONS  AND  RECOMMENDATIONS 

A.  MEASURING  COMPUTER  SYSTEM  EFFECTIVENESS 

In  an  environment  of  scarce  financial  resources, 
organizations  need  to  measure  the  effectiveness  of  computer 
systems  to  justify  expenditures.  Of  the  several  direct  and 
in-direct  methods  available  to  measure  computer  system 
effectiveness,  measurement  of  user  information  satisfaction  is 
a  suitable,  available  and  effective  surrogate  measure.  The 
previously  developed  and  validated  user  information 
satisfaction  questionnaire  of  Baroudi  and  Orlikowski  (1988)  is 
easily  administered  and  provides  sufficient  responses  to 
determine  a  baseline  reference  point  for  overall  satisfaction 
as  well  as  indicating  potential  problem  areas.  The  survey  is 
not  a  definitive  evaluation  of  CHCS,  but  combined  with  further 
investigation,  can  be  a  powerful  tool  in  the  analysis  and 
interpretation  into  the  cause  of  user  dissatisfaction. 

B.  SATISFACTION  CHARACTERISTICS  OF  CHCS 

Overall  satisfaction  is  a  generalization  of  all  the 
characteristics  that  effect  the  satisfaction  of  the  user. 
Along  with  overall  satisfaction,  the  study  specifically  looked 
at  the  four  factors  that  make  up  satisfaction:  local  MID  staff 
and  services,  contractor  services,  knowledge  and  involvement, 
and  information  product. 
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1 .  Overall  Satisfaction 

The  survey  results  indicate  the  users  are  slightly 
satisfied  with  CHCS  at  Naval  Hospital,  Charleston  as  a  whole. 
The  overall  satisfaction  scores  do  not  differ  significantly 
for  the  various  types  of  personnel  operating  the  system.  In 
other  words,  the  different  ways  (modules)  in  which  people  use 
the  system  did  not  affect  their  overall  perception  of  the 
system. 

2 .  Areas  of  Satisfaction 
All  three  work  groups  rated  the  local  MID  staff  and 

services  as  quite  satisfactory.  This  is  an  indication  that 
the  local  MID  personnel  is  interacting  well  with  the  system 
users.  The  users  perceive  the  local  MID  is  taking  care  of 
their  needs . 

The  output  of  the  system,  or  information  product,  is 
considered  by  the  users  as  slightly  satisfactory  overall. 
What  they  receive  from  the  system  in  the  form  of  output  meets 
their  expectations. 

Although  the  overall  satisfaction  scores  did  not 
differ  significantly,  the  administrative  work  group  displayed 
higher  satisfaction  in  many  areas.  The  slightly  higher 
satisfaction  can  be  traced  to  their  higher  perception  of 
satisfaction  with  the  training  provided.  This  leads  to  higher 
satisfaction  in  participation  and  understanding  of  the  system. 
The  manner  in  which  the  administrative  group  personnel  are 
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trained  or  assisted  is  producing  a  more  positive  effect  on 
their  satisfaction  than  the  other  work  groups. 

3.  Areas  of  Dissatisfaction 

Factors  that  result  in  user  dissatisfaction  or 
significantly  lower  satisfaction  when  compared  to  other 
factors  are  potential  areas  for  concern.  Further 
investigation  into  areas  of  dissatisfaction  can  provide 
context,  history  and  insight  for  possible  corrective  measures. 

The  respondents  were  most  dissastified  with  the 
contractor  services  in  respect  to  the  time  for  new  system 
developement  and  changes  to  the  system.  Although  the  exact 
cause  of  this  dissatisfaction  cannot  be  determined,  most 
likely,  the  user' s  perception  of  the  time  required  for  these 
contractor  activities  is  much  less  than  the  reality  of  the 
situation. 

4 .  Satisfaction  with  Time  of  System  Use 

The  expectation  is  that  users  will  become  more 
comfortable  and  confident,  and  therefore  more  satisfied,  with 
CHCS  as  they  gain  increased  experince  using  the  system.  This 
expectation  was  shown  true  for  the  physician  and  the 
administrative  groups.  However,  the  ancillary  work  group's 
satisfaction  with  the  system  was  lower  for  experienced  users 
than  the  less  experienced  users. 
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C .   RECOMMENDAT I ONS 

This  study  showed  that  the  short- form  questionnaire 
developed  by  Baroudi  and  Orlikowski  (1988)  is  an  effective 
means  to  measure  user  information  satisfaction.  The  survey 
instrument  is  an  appropriate  instrument  to  document  user 
satisfaction  with  the  CHCS  at  other  military  medical 
facilities  as  well  as  documenting  changes  in  user  satisfaction 
at  Naval  Hospital ,    Charleston  at  a  later  time. 

A  survey  of  this  type  is  best  conducted  by  an  individual 
at  the  site.  It  is  beneficial  for  the  researcher  to  have 
first  hand  experience  as  a  user  of  CHCS.  Additionally,  direct 
contact  with  individuals  for  interview  by  the  researcher  is 
beneficial  for  adding  context  to  history. 

Additional  investigation  should  be  undertaken  in  several 
areas  to  add  context  and  history.  A  larger  and  clearer 
picture  may  explain  the  reasons  why  the  work  groups  had 
significant  differences  in  their  reported  satisfaction  to  the 
various  factors.   These  include: 

1.  The  administrative  group's  overall  higher  satisfaction 
over  the  other  work  groups 

2.  The  physicians'  lowered  perception  of  satisfaction  in 
almost  all  areas 

3.  The   reasons   for  the   respondents   having  negative 
perceptions  of  satisfaction  with  contractor's  services 

4.  The   reduction   of   satisfaction   for   experienced 
ancillary  users  from  less  experienced  ancillary  users 
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The  Baroudi  and  Orlikowski  short-form  questionnaire  (1988) 
should  be  used  to  measure  UIS  at  other  CHCS  sites  for 
comparison  to  the  results  obtained  from  this  study  at  Naval 
Hospital,  Charleston.  The  survey  should  also  be  conducted  at 
a  later  time  at  Naval  Hospital,  Charleston  to  monitor  for  any 
changes  in  UIS. 
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APPENDIX  A 
Part.  A:  General  Information 

1.   Hospital  Department:  (Check  one) 

General  Administration 

Nursing  Administration 

Dietary 

Emergency 

Laboratory 

Medical  Clinic 

Inpatient  Nursing 

Pharmacy 

Radiology 

Other  (Specify) : 


2.   Job  Description:  (Check  one) 

Corpsman  (0000) 

Technician 

Nurse 

Pharmacist 

Physician 

Physician  Assistant 

Other  (Specify) : 


3.   Highest  Level  of  Education:  (Check  one) 

High  School  Graduate 

Some  College 

Bachelor's  Degree 

Some  Graduate  Work 

Master's  Degree 

Doctoral  Degree 

Medical  Degree 

Other  (Specify) : 


4 .   Age 


5.  Gender:  Male  Female 

6.  Length  of  time  (in  months)  you  have  used  CHCS : 


7.   Have  you  used  other  computer  systems  before  ?   Yes 


8.   If  your  answer  was  Yes  to  question  7,  was  it  a  health  care 
information  system  ?   Yes  No 
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Part  B:  The  Questionnaire 

This  section  of  the  survey  conveys  your  own  personal  feelings 
concerning  the  use  of  the  Composite  Health  Care  system  at  Naval 
Hospital,  Charleston.   Please  do  not  attempt  to  analyze  the 
questions.  Remember,  there  are  no  right  or  wrong  answers. 

Please  follow  these  instructions: 

1.  Check  each  scale  in  the  position  that  describes  your  evaluation 
of  the  factor  being  described. 

2.  Check  each  scale,  do  not  omit  any. 

3.  Check  only  one  position  for  each  scale. 

4.  Check  in  the  space,  not  between  spaces.       THIS,  NOT  THIS 

;  X  :    X    : 

5.  Work  rapidly.   Rely  on  your  first  impressions. 

ANSWERS  BASED  ON  YOUR  OWN  FEELINGS 
Relationship  with  the  Management  Information  Department  (MID)  staff 

dissonant  : : : : : : : :  harmonious 

bad  : : : : : : : :  good 

Processing  of  requests  for  changes  to  existing  systems 

fast  : : : : : : : :  slow 

untimely  : : : : : : : :  timely 

Degree  of  training  provided  to  users 

complete  : : : : : : : :  incomplete 

low  : : : : : : : :  high 

User' s  understanding  of  systems 

insufficient  : : : : : : : :  sufficient 

complete  : : : : : : : :  incomplete 

User's  feeling  of  participation 

positive  : : : : : : : :  negative 

insufficient  ::::::::  sufficient 
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6.  Attitude  of  the  Management  Information  Department  staff 

cooperative  : : : : : : : :  belligerent 

negative  : : : : : : : :  positive 

7 .  Reliability  of  output  information 

high  : : : : : : : :  low 

superior  : : : : : : : :  inferior 

8 .  Relevancy  of  output  information  (to  intended  function) 

useful  : : : : : : : :  useless 

relevant  : : : : : : : :  irrelevant 

9 .  Accuracy  of  output  information 

inaccurate  : : : : : : : :  accurate 

low  : : : : : : : :  high 

10.  Precision  of  output  information 

low  : : : : : : : :  high 

definite  : : : : : : : :  uncertain 

11.  Communication  with  the  Management  Information  Department  staff 

dissonant  : : : : : : : :  harmonious 

destructive  : : : : : : : :  productive 

12 .  Time  required  for  new  systems  development 
unreasonable  : : : : : : : :  reasonable 

acceptable  : : : : : : : :  unacceptable 

13 .  Completeness  of  the  output  information 

sufficient  : : : : : : : :  insufficient 

adequate  : : : : : : : :  inadequate 

Thanks  again  for  your  cooperation 
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APPENDIX  B 


Composite  Health  Care  System 
User  Satisfaction  Survey 


This  survey  is  part  of  a  study  of  the  Composite  Health  Care  System 
(CHCS)  effectiveness  in  military  hospital  inpatient,  outpatient  and 
administrative  settings.   The  purpose  of  the  study  is  to  obtain  information 
about  your  perceptions  of  how  well  CHCS  functions  in  your  area. 

Just  a  few  minutes  are  required  to  fill  out  the  brief  general 
information  and  the  13  question  survey.   Your  responses  will  be  treated  in 
complete  confidentiality.   The  data  gathered  through  the  survey  will 
provide  valuable  insights  into  the  system's  strengths  and  weaknesses,  and 
assist  in  future  development,  training  and  use. 

The  success  of  this  survey  depends  on  receiving  as  many  completed 
surveys  as  possible  from  users  in  all  hospital  areas.   Your  participation 
makes  an  important  contribution  towards  this  end.   There  are  no  right  of 
wrong  answers,  only  your  candid  response  to  each  question.   Please  do  not 
omit  any  of  the  questions  in  either  section  of  the  survey. 

Please  return  your  completed  survey  to  LT  Powell  in  the  Pharmacy 
Department  in  the  addressed  envelope  provided.   Thank  you  for  your 
cooperation. 


LT  L.E.  Hurd,  MSC 

Naval  Postgraduate  School 

Monterey,  Ca 
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APPENDIX  C-l 


COMBINED  RESULTS  FOR  DOCTORS  VS  ADMINISTRATIVE  VS  ANCILLARY 


SATISFACTION 

AVERAGE 

INDEX 

DOCTOR 

0.620879 

8.07 

ADMIN 

1.183258 

14.96 

ANCILLARY 

0.742475 

9.65 

INDIVIDUAL  QUESTION  RESPONSES 


QUESTION          12         3         4  5 

DOCTOR    1.380952  -0.45238  0.047619  0.142857  0.52381 

ADMIN     1.897059  -0.08824  1.426471  1.411765  1.588235 

ANCILARY  1.652174  -0.82609  0.184783  1.032609  0.913043 

STAND  DEV. 

DOCTOR     1.28694  1.328898  1.342593  1.601393  1.611515 

ADMIN     1.198752  1.967916  1.586474  1.498609  1.30601 

ANCILARY  1.287597  1.700735  1.803107  1.712598  1.691827 


QUESTION  6  7         8         9  10 

DOCTOR  1.571429  1.071429  0.571429  1.404762  0.857143 

ADMIN  2.308824  1.029412  1.220588  1.117647  0.808824 

ANCILARY  1.934783  0.48913  0.902174  1.097826  0.206522 

STAND  DEV. 

DOCTOR  1.085219  0.921104  1.399353  1.083345  0.92582 

ADMIN  0.965948  1.392631  1.401977  1.766432  1.650485 

ANCILARY  1.014232  1.72566  1.437793  1.576334  1.56569 


QUESTION          11  12  13 

DOCTOR  1.357143  -0.92857  0.52381 

ADMIN  1.941176  -0.07353  0.794118 

ANCILARY  1.576087  0.01087  0.478261 

STAND  DEV. 

DOCTOR  1.185716  1.471566  0.833391 

ADMIN  1.144518  1.764629  1.550723 

ANCILARY  1.178877  1.775874  1.660652 
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APPENDIX  C-2 


PHYSICIANS  vs  THE  REST  OF  THE  SURVEY  GROUP 


GENERAL 


AVERAGE 


INDEX 


DOCTOR  SAT 

0.620879 

8.07 

STDEV 

1.445 

8.37 

NON-DOC  SAT 

0.930288 

12.09 

STDEV 

1.703 

12.44 

GROUPED  FACTORS 


DOCTOR  GROUP 
NON-DOC  GROUP 
DOC  STDEV 
NON-DOC  STDEV 


A 

A 

(MID) 

(CONT) 

1.44 

-0.27 

1.86 

-0.69 

1.17 

1.82 

1.16 

1.41 

0.885714  0.065476 

0.761773  0.65 

1.09207  1.505041 

1.604875  1.856174 


QUESTION 


DOCTOR  SATIS 
NON-DOC  SATIS 
DOC  STDEV 
NON-DOC  STDEV 


1.380952  -0.45238  0.047619  0.142857   0.52381 

1.75625  -0.5125  0.7125   1.19375   1.20625 

1.28694  1.328898  1.342593  1.601393  1.611515 

1.252654  1.849656  1.81672  1.631054  1.578539 


QUESTION 


8 


10 


DOCTOR  SATIS 
NON-DOC  SATIS 
DOC  STDEV 
NON-DOC  STDEV 


1.571429  1.071429  0.571429  1.404762  0.857143 


2.09375   0.71875 
1.085219  0.921104  1 
1.008123   1.61039  1 


1.0375  1.10625  0.4625 
399353  1.083345  0.92582 
426997  1.654409  1.624837 


QUESTION 


11 


12 


13 


DOCTOR  SATIS 
NON-DOC  SATIS 
DOC  STDEV 
NON-DOC  STDEV 


1.357143  -0.92857  0.52381 

1.73125  -0.025  0.6125 

1.185716  1.471566  0.833391 

1.174798  1.766032  1.617466 


71 


APPENDIX  C-3 

PHYSICIANS  vs  THE  REST  OF  THE  STUDY  GROUP 
CALCULATED  T -VALUE 
QUESTION  T -VALUE 


1 

-1.196 

2 

0.169 

3 

-1.869 

4 

-2.666 

5 

-1.734 

6 

-1.991 

7 

1.313 

8 

-1.353 

9 

0.998 

10 

1.459 

11 

-1.289 

12 

-2.399 

13 

-0.241 

FACTOR 

A 

-1.467 

MID 

A 

0.982 

CONTRACTOR 

B 

-0.334 

C 

1.331 

OVERALL      1.397 


*  SIGNIFICANT  AT  ALPHA  =0.05 
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APPENDIX  C-4 


ANCILLARY  vs  THE  REST  OF  THE  SURVEY  GROUP 


GENERAL 


AVERAGE 


INDEX 


ANCIL  SATIS 

0.742475 

9.65 

NON-ANCIL  SATIS 

0.967832 

12.6 

ANCIL  STDEV 

1.722284 

11.7 

NON-ANCIL  STDEV 

1.595243 

11.8 

GROUPED  FACTORS 


ANCIL  GROUP 
NON-ANCI  GROUP 
ANCIL  STDEV 
NON-ANCI  STDEV 


A 

A 

(MID) 

(CONT) 

1.72 

-0.41 

1.82 

-0.32 

1.17 

1.78 

1.18 

1.73 

B 


0.634783  0.328804 

0.935614  0.697727 

1.621374  1.874259 

1.395514  1.725329 


QUESTION 

ANCILLARY 
NON- ANCILLARY 
ANC  ST  DEV 


1.652174  -0.82609  0 

1.7  -0.22727 
1.287597  1.700735  1 


184783  1.032609  0.913043 

0.9  0.927273  1.181818 

803107  1.712598  1.691827 


NON-ANC  ST  DEV  1.252887  1.753942  1.636454   1.65185  1.527616 


QUESTION 

ANCILLARY  1 

NON-ANC I LLARY  2 

ANC  ST  DEV  1 


8 


10 


934783  0.48913  0.902174  1.097826  0.206522 

027273  1.045455  0.972727  1.227273  0.827273 

014232  1.72566  1.437793  1.576334  1.56569 

NON-ANC  ST  DEV  1.070532  1.229502  1.430077  1.542422  1.413299 


QUESTION 


11 


12 


13 


ANCILLARY 
NON-ANC I LLARY 
ANC  ST  DEV 


1.576087  0.01087  0.478261 
1.718182  -0.40909  0.690909 
1.178877  1.775874  1.660652 


NON-ANC  ST  DEV  1.189648   1.70426  1.325442 
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APPENDIX  C-5 

ANCILLARY  vs  THE  REST  OF  THE  STUDY  GROUP 
CALCULATED  T -VALUE 

QUESTION   T -VALUE 


1 

-0.18 

2 

-1.653 

3 

-1.98 

4 

0.299 

5 

-0.795 

6 

-0.423 

7 

-1.768 

8 

-0.235 

9 

-0.396 

10 

-1.984 

11 

-0.572 

12 

1.151 

13 

-0.674 

FACTOR 

A 

-0.406 

MID 

A 

-0.245 

CONTRACTOR 

B 

-0.948 

C 

-0.976 

OVERALL     -1.197 

*  SIGNIFICANT  AT  ALPHA  =0.05 
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APPENDIX  C-6 


ADMINISTRATION  vs  THE  REST  OF  THE  SURVEY  GROUP 


GENERAL 


AVERAGE 


INDEX 


ADMIN  SAT 

1.182127 

14.96 

NON-ADMIN  SAT 

0.704363 

12.94 

ADMIN  STDEV 

1.6444 

9.16 

NON-ADMIN  STDEV 

1.645581 

10.72 

GROUPED  FACTORS 


ADMIN  GROUP 
NON-ADMIN  GROUP 
ADMIN  ST  DEV 
NON-ADMIN  ST  DEV 


A 

A 

(MID) 

(CONT) 

2.05 

-0.09 

1.63 

-0.8 

1.12 

1.86 

1.18 

1.68 

0.994118  1.084559 

0.664463  0.244403 

1.559052  1.742729 

1.486632  1.770465 


QUESTION 

1 

2 

3 

4 

5 

ADMIN            1 
NON -ADMIN       1 
ADMIN  STDEV     1 
NON-ADMIN  STDEV1 

.897059 
.567164 
.198752 
.265863 

-0.08824 
-0.70896 
1.967916 
1.751332 

1 
0 
1 

1 

.426471 
.141791 
.586474 
.746904 

1 
0 
1 
1 

.411765 
.753731 
.498609 
.676404 

1.588235 

0.791045 

1.30601 

1.605592 

QUESTION 

6 

7 

8 

9 

10 

ADMIN           2.308824 
NON-ADMIN       1.820896 
ADMIN  STDEV     0.96594  8 
NON-ADMIN  STDEV  1.04371 

1.029412 
0.671642 
1.392631 
1.498369 

1 
0 

1 
1 

.220588 
.798507 
.401977 
.430456 

1.117647 

1.19403 

1.766432 

1.555407 

0 
0 
1 
1. 

808824 
410448 
650485 
512961 

QUESTION 

11 

12 

13 

ADMIN 
NON-ADMIN 
ADMIN  STDEV 

1 
1 
1 

.941176 
.507463 
.144518 

-0.08824 
-0.28358 
1.764629 

0 
0 

1 

.794118 
.492537 
.550723 

NON-ADMIN  STDEV1. 183936   1.74471  1.487448 
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APPENDIX  C-7 

ADMINISTRATION  vs  THE  REST  OF  THE  STUDY  GROUP 
CALCULATED  T -VALUE 

QUESTION   T -VALUE 


1 

1.259 

2 

1.614 

3 

3.599  * 

4 

1.929 

5 

2.503  * 

6 

2.275  * 

7 

1.161 

8 

1.411 

9 

-0.221 

10 

1.213 

11 

1.759 

12 

0.53 

13 

0.949 

FACTOR 

A 

1.719 

MID 

A 

1.935 

CONTRACTOR 

B 

1.036 

C 

2.265  * 

OVERALL      0.938 

*  SIGNIFICANT  AT  ALPHA  =0.05 
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APPENDIX  C-8 

SATISFACTION  BASED  ON  TIME  ON  THE  SYSTEM 

OVERALL 

TIME  OF 
#  PEOPLE  USE       AVG  SAT 


28 

1  TO  5 

11.64286 

36 

6  TO  11 

11.84722 

37 

>=  12 

10.37838 

101 

ADMINISTRATION 

TIME  OF 

#  PEOPLE  USE       AVG  SAT 

8  1  TO  5  8.8125 

9  6  TO  11  17.4375 
17  >=  12  17.36111 
34 


ANCILLARY 

TIME  OF 

#  PEOPLE  USE      AVG  SAT 

12  1  TO  5    16.91667 

19  6  TO  11       11.5 

15  >=  12     1.533333 
46 


DOCTORS 

#  PEOPLE 

8 

8 

5 

21 


TIME  OF 

USE 

AVG  SAT 

1  TO  5 

6.5625 

6  TO  11 

6.375 

>=12 

13.2 
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